Apparatus and method of estimating values from images

ABSTRACT

A method is used to generate a distortion model for a structured illumination microscopy (SIM) optical system. A sliding window is moved in relation to a plurality of images to define a plurality of sub-tiles. Each sub-tile represents a portion of the corresponding image. Parameters are estimated for each sub-tiles. The parameters include two or more parameters selected from the group consisting of modulation, angle, spacing, phase offset, and phase deviation. A full width at half maximum (FWHM) value associated with each sub-tile is estimated. A distortion model is estimated, based at least in part on a combination of the estimated parameters and FWHM values stored in the predetermined format and an estimated center window parameter. A two-dimensional image may be generated, based at least in part on the estimated distortion model. The two-dimensional image may include representations indicating where distortions occur in the optical system.

RELATED APPLICATIONS

This application claims priority to U.S. Provisional Pat. App. No.62/944,687, entitled “Apparatus and Method of Estimating Values fromImages,” filed Dec. 6, 2019, the disclosure of which is incorporated byreference herein, in its entirety.

BACKGROUND

The subject matter discussed in this section should not be assumed to beprior art merely as a result of its mention in this section. Similarly,a problem mentioned in this section or associated with the subjectmatter provided as background should not be assumed to have beenpreviously recognized in the prior art. The subject matter in thissection merely represents different approaches, which in and ofthemselves may also correspond to implementations of the claimedtechnology.

Structured illumination microscopy (SIM) is a class of computationalimaging algorithm that reconstructs super resolution images frommultiple lower-resolution source images. To ensure successfulreconstruction, the source raw images should be of high quality. Highquality raw images require careful tuning, calibration, and assessmentof the optics performance of the imaging instrument. In addition toconventional imaging instrument characterization, the SIM imaging opticshave additional components that need to be further characterized andvalidated.

SUMMARY

It may be desirable to provide systems and methods for promoting qualitycontrol and calibration with imaging optics and associated opticalcomponents within a SIM system, particularly a SIM system that is usedfor imaging biological samples such as nucleotide sequences. Describedherein are devices, systems, and methods for processing images capturedusing SIM to overcome the pre-existing challenges and achieve thebenefits as described herein.

An implementation relates to a method that includes receiving aplurality of images captured using structured illumination microscopy(SIM) in an optical system, each image of the plurality of images havinga first field of view. The method further includes defining a window,the window defining a second field of view representing a portion of thefirst field of view such that the second field of view is smaller thanthe first field of view. The method further includes moving the windowin relation to each image of a plurality of images. The method furtherincludes capturing a plurality of sub-tiles from each image of theplurality of images while moving the window in relation to each image ofthe plurality of images, each sub-tile of the plurality of plurality ofsub-tiles representing a portion of the corresponding image of theplurality of images, the portion represented by each sub-tile of theplurality of sub-tiles being defined by the second field of view at aposition corresponding to a moment at which the sub-tile of theplurality of sub-tiles is captured. The method further includesestimating parameters associated with each sub-tile of the plurality ofsub-tiles, the parameters including two or more parameters selected fromthe group consisting of modulation, angle, spacing, phase offset, andphase deviation. The method further includes estimating a full width athalf maximum (FWHM) value associated with each sub-tile of the pluralityof sub-tiles. The method further includes storing the estimatedparameters and FWHM values in a predetermined format.

In some implementations of a method, such as that described in thepreceding paragraph of this summary, the plurality of images comprisingtwelve images.

In some implementations of a method, such as any of those described inany of the preceding paragraphs of this summary, the plurality of imagesincluding a first set of images associated with a first color and asecond set of images associated with a second color.

In some implementations of a method, such as any of those described inany of the preceding paragraphs of this summary, the plurality of imagesincluding a first set of images associated with a first gratingorientation and a second set of images associated with a second gratingorientation.

In some implementations of a method, such as that described in thepreceding paragraph of this summary, the method further includescapturing the plurality of images. The method further includes, whilecapturing the plurality of images, moving a light source relative to oneor more phase masks from a first position to a second position, thefirst position providing the first grating orientation and the secondposition providing the second grating orientation.

In some implementations of a method, such as any of those described inany of the preceding paragraphs of this summary, the plurality of imagesincluding a first set of images associated with a first phase and asecond set of images associated with a second phase.

In some implementations of a method, such as that described in thepreceding paragraph of this summary, the method further includescapturing the plurality of images. The method further includes, whilecapturing the plurality of images, moving a reflective element from afirst position to a second position, the first position providing thefirst phase and the second position providing the second phase.

In some implementations of a method, such as any of those described inany of the preceding paragraphs of this summary, the predeterminedformat comprises a table.

In some implementations of a method, such as that described in thepreceding paragraph of this summary, the table is in the form of atwo-dimensional table.

In some implementations of a method, such as any of those described inany of the preceding paragraphs of this summary, the method furtherincludes estimating a center window parameter, the center windowparameter corresponding to a central region within the first field ofview.

In some implementations of a method, such as that described in thepreceding paragraph of this summary, the method further includesestimating a distortion model based at least in part on a combination ofthe estimated parameters and FWHM values stored in the predeterminedformat and the estimated center window parameter.

In some implementations of a method, such as that described in thepreceding paragraph of this summary, the estimating the distortion modelincluding subtracting the estimated center window parameter from theestimated parameters and FWHM values stored in the predetermined format.

In some implementations of a method, such as that described in thepreceding paragraph of this summary, the estimating the distortion modelincluding fitting a quadratic surface function to the result of thesubtracting.

In some implementations of a method, such as that described in thepreceding paragraph of this summary, the fitting the quadratic surfacefunction to the result of the subtracting comprising using a shrinkageestimator.

In some implementations of a method, such as any of those described inany of the preceding four paragraphs of this summary, the method furtherincludes validating the estimated distortion model by calculating acoefficient of determination for the estimated distortion model andcomparing the calculated coefficient of determination to a predeterminedthreshold value.

In some implementations of a method, such as any of those described inany of the preceding five paragraphs of this summary, the method furtherincludes estimating a phase offset and applying the phase offset to theestimated distortion model.

In some implementations of a method, such as any of those described inany of the preceding six paragraphs of this summary, the method furtherincludes generating a two-dimensional image based at least in part onthe estimated distortion model, the two-dimensional image includingrepresentations indicating where distortions occur in the opticalsystem.

In some implementations of a method, such as any of those described inany of the preceding seven paragraphs of this summary, the methodfurther includes capturing a subsequent plurality of images using SIM inthe optical system. The method further includes generating ahigh-resolution image based at least in part on the plurality of images,the generating the high-resolution image including adjusting data fromthe subsequent plurality of images based at least in part on theestimated distortion model.

In some implementations of a method, such as any of those described inany of the preceding paragraphs of this summary, the method furtherincludes capturing a subsequent plurality of images using SIM in theoptical system. The method further includes generating a high-resolutionimage based at least in part on the plurality of images, the generatingthe high-resolution image including adjusting data from the subsequentplurality of images based at least in part on the estimated parametersand FWHM values stored in the predetermined format.

In some implementations of a method, such as any of those described inany of the preceding two paragraphs of this summary, the subsequentplurality of images including images of nucleotides.

In some implementations of a method, such as any of those described inany of the preceding paragraphs of this summary, the method furtherincludes capturing the plurality of images using SIM in the opticalsystem, the received plurality of images including the capturedplurality of images.

In some implementations of a method, such as that described in thepreceding paragraph of this summary, the plurality of captured imagesare images of an optical target.

In some implementations of a method, such as that described in thepreceding paragraph of this summary, the optical target includes a dye,and the capturing of the plurality of images includes exciting moleculesin the dye.

In some implementations of a method, such as that described in thepreceding paragraph of this summary, the dye has a mean emissionwavelength, and exciting molecules in the dye includes emitting anexcitation light toward the dye, the excitation light having awavelength that is substantially longer than the mean emissionwavelength of the dye.

In some implementations of a method, such as that described in thepreceding paragraph of this summary, the dye includes Coumarin dye.

In some implementations of a method, such as that described in thepreceding paragraph of this summary, the excitation light has awavelength of at least approximately 520 nm.

In some implementations of a method, such as that described in thepreceding paragraph of this summary, the method further includesobserving green laser generated fringes at blue wavelengths.

In some implementations, an apparatus includes a first optical assemblyto emit structured illumination toward a target. The first opticalassembly includes a light emitting assembly, a first phase mask toimpart a first pattern to light emitted by the light emitting assembly,a second phase mask to impart a second pattern to light emitted by thelight emitting assembly, and a phase adjustment assembly to adjust aphase of light structured by the first phase mask and the second phasemask. The apparatus further includes a second optical assembly. Thesecond optical assembly includes an image sensor to capture images ofthe target as illuminated by the first optical assembly. The apparatusfurther includes a processor. The processor is to receive a plurality ofimages captured using the image sensor, each image of the plurality ofimages having a first field of view. The processor is further to definea window, the window defining a second field of view representing aportion of the first field of view such that the second field of view issmaller than the first field of view. The processor is further to movethe window in relation to each image of a plurality of images. Theprocessor is further to capture a plurality of sub-tiles from each imageof the plurality of images while moving the window in relation to eachimage of the plurality of images, each sub-tile of the plurality ofplurality of sub-tiles representing a portion of the corresponding imageof the plurality of images, the portion represented by each sub-tile ofthe plurality of sub-tiles being defined by the second field of view ata position corresponding to a moment at which the sub-tile of theplurality of sub-tiles is captured. The processor is further to estimateparameters associated with each sub-tile of the plurality of sub-tiles,the parameters including two or more parameters selected from the groupconsisting of modulation, angle, spacing, phase offset, and phasedeviation. The processor is further to estimate a full width at halfmaximum (FWHM) value associated with each sub-tile of the plurality ofsub-tiles. The processor is further to store the estimated parametersand FWHM values in a predetermined format.

In some implementations of an apparatus, such as that described in thepreceding paragraph of this summary, the target includes a samplecontainer.

In some implementations of an apparatus, such as that described in thepreceding paragraph of this summary, the target includes a biologicalsample in the sample container.

In some implementations of an apparatus, such as any of those describedin any of the preceding paragraphs of this summary, the light emittingassembly is to emit light in at least two channels.

In some implementations of an apparatus, such as that described in thepreceding paragraph of this summary, the at least two channels includingat least two colors, each color of the at least two colors beingcorresponding to a corresponding channel of the at least two channels.

In some implementations of an apparatus, such as any of those describedin any of the preceding paragraphs of this summary, the first opticalassembly further includes a grating switcher. The grating switcher is toselectively direct or permit light emitted from the light emittingassembly toward the first phase mask or the second phase mask.

In some implementations of an apparatus, such as that described in thepreceding paragraph of this summary, the grating switcher includes atleast one movable reflective element.

In some implementations of an apparatus, such as that described in thepreceding paragraph of this summary, the grating switcher furtherincludes a rotatable plate supporting the movable reflective element.The rotatable plate is rotatable to thereby selectively position thereflective element in relation to the first phase mask or the secondphase mask, to thereby selectively direct or permit light emitted fromthe light emitting assembly toward the first phase mask or the secondphase mask.

In some implementations of an apparatus, such as any of those describedin any of the preceding paragraphs of this summary, the phase adjustmentassembly includes a movable reflecting element.

In some implementations of an apparatus, such as that described in thepreceding paragraph of this summary, the phase adjustment assemblyfurther includes an actuator to move the movable reflecting element.

In some implementations of an apparatus, such as that described in thepreceding paragraph of this summary, the actuator is to move the movablereflecting element along a linear path.

In some implementations of an apparatus, such as any of those describedin any of the two preceding paragraphs of this summary, the actuatorincludes a piezoelectric element.

In some implementations of an apparatus, such as any of those describedin any of the preceding paragraphs of this summary, the processor isfurther to estimate a center window parameter, the center windowparameter corresponding to a central region within the first field ofview.

In some implementations of an apparatus, such as that described in thepreceding paragraph of this summary, the processor is further toestimate a distortion model based at least in part on a combination ofthe estimated parameters and FWHM values stored in the predeterminedformat and the estimated center window parameter.

In some implementations of an apparatus, such as that described in thepreceding paragraph of this summary, the processor is further toestimate the distortion model by including subtracting the estimatedcenter window parameter from the estimated parameters and FWHM valuesstored in the predetermined format.

In some implementations of an apparatus, such as that described in thepreceding paragraph of this summary, the processor is further toestimate the distortion mode by fitting a quadratic surface function tothe result of the subtracting.

In some implementations of an apparatus, such as that described in thepreceding paragraph of this summary, the processor is further toestimate the distortion mode by fitting the quadratic surface functionto the result of the subtracting comprising using a shrinkage estimator.

In some implementations of an apparatus, such as any of those describedin any of the four preceding paragraphs of this summary, the processoris further to validate the distortion model by calculating a coefficientof determination for the estimated distortion model, and comparing thecalculated coefficient of determination to a predetermined thresholdvalue.

In some implementations of an apparatus, such as any of those describedin any of the five preceding paragraphs of this summary, the processoris further to estimate a phase offset, and apply the phase offset to theestimated distortion model.

In some implementations of an apparatus, such as any of those describedin any of the six preceding paragraphs of this summary, the processor isfurther to generate a two-dimensional image based at least in part onthe estimated distortion model, the two-dimensional image includingrepresentations indicating where distortions occur in the opticalsystem.

In some implementations of an apparatus, such as any of those describedin any of the six preceding paragraphs of this summary, the processor isfurther to capture a subsequent plurality of images using SIM in theoptical system, and generate a high-resolution image based at least inpart on the plurality of images. Generating the high-resolution imageincludes adjusting data from the subsequent plurality of images based atleast in part on the estimated distortion model.

In some implementations of an apparatus, such as any of those describedin any of the preceding paragraphs of this summary, the processor isfurther to capture a subsequent plurality of images using SIM in theoptical system, and generate a high-resolution image based at least inpart on the plurality of images. Generating the high-resolution imageincluding adjusting data from the subsequent plurality of images basedat least in part on the estimated parameters and FWHM values stored inthe predetermined format.

In some implementations, a method includes capturing a plurality ofimages using structured illumination microscopy (SIM) in an opticalsystem. The plurality of captured images are images of an opticaltarget. The optical target includes a dye. Capturing the plurality ofimages includes exciting molecules in the dye. The dye has a meanemission wavelength. Exciting molecules in the dye includes emitting anexcitation light toward the dye. The excitation light has a wavelengththat is substantially longer than the mean emission wavelength of thedye. The method further includes observing fringes in the plurality ofimages. The observed fringes are at a first wavelength associated with afirst color. The observed fringes are generated by a light sourceemitting light at a second wavelength associated with a second color.

In some implementations of a method, such as that described in thepreceding paragraph of this summary, the dye includes Coumarin dye.

In some implementations of a method, such as that described in thepreceding paragraph of this summary, the excitation light has awavelength of at least approximately 520 nm.

In some implementations of a method, such as any of those described inany of the three preceding paragraphs of this summary, the first coloris blue and the second color is green.

In some implementations, a processor-readable medium includes contentsthat are configured to cause a computing system to process data byperforming the method of any one or more of the methods described in anyof the preceding paragraphs of this summary.

It should be appreciated that all combinations of the foregoing conceptsand additional concepts discussed in greater detail below (provided suchconcepts are not mutually inconsistent) are contemplated as being partof the inventive subject matter disclosed herein and to achieve thebenefits/advantages as described herein. In particular, all combinationsof claimed subject matter appearing at the end of this disclosure arecontemplated as being part of the inventive subject matter disclosedherein.

BRIEF DESCRIPTION OF THE DRAWINGS

The details of one or more implementations are set forth in theaccompanying drawings and the description below. Other features,aspects, and advantages will become apparent from the description, thedrawings, and the claims, in which:

FIG. 1A depicts an example of a Moire fringe formation by using agrating with one-dimensional (1D) modulation.

FIG. 1B depicts a graphical illustration of illumination intensitiesproduced by a two-dimensional (2D) structured illumination pattern.

FIG. 1C depicts an example of a geometrical pattern for a nanowellarrangement.

FIG. 2 depicts a schematic diagram of a SIM biological sample imagingsystem that may utilize spatially structured excitation light to image asample.

FIG. 3 depicts a schematic diagram of an example of an alternativeoptical assembly for use in the SIM biological sample imaging system ofFIG. 2 .

FIG. 4 depicts a schematic diagram of a phase mask assembly of theoptical assembly of FIG. 3 .

FIG. 5A depicts a schematic diagram of the optical assembly of FIG. 3with a grating switcher in a first state and an adjustable reflectingelement in a first state.

FIG. 5B depicts a schematic diagram of the optical assembly of FIG. 3with the grating switcher in the first state and the adjustablereflecting element in a second state.

FIG. 5C a schematic diagram of the optical assembly of FIG. 3 with thegrating switcher in a second state and the adjustable reflecting elementin the first state.

FIG. 5D a schematic diagram of the optical assembly of FIG. 3 with thegrating switcher in the second state and the adjustable reflectingelement in the second state.

FIG. 6A is a simplified depiction of bending parallel lines due todistortion of a lens that magnifies.

FIG. 6B illustrates a first set of measurements made to wavelengths ofspacing between nominally parallel lines.

FIG. 6C depicts a second set of measurements made to wavelengths ofspacing between nominally parallel lines.

FIG. 6D depicts an example of sub-tiles or sub-fields of a full field ofview (FOV) image.

FIG. 7 depicts a flow chart of an example of a process for generatingand applying a distortion model in SIM imaging.

FIGS. 8A-8C depict a schematic view of a sliding window traversing animage.

FIG. 9 depicts a flow chart showing an example of a process forestimating distortion models.

FIG. 10 depicts a schematic illustration of Anti-Stokes emission.

It will be recognized that some or all of the figures are schematicrepresentations for purposes of illustration. The figures are providedfor the purpose of illustrating one or more implementations with theexplicit understanding that they will not be used to limit the scope orthe meaning of the claims.

DETAILED DESCRIPTION

In some aspects, methods and systems are disclosed herein for promotingquality control and calibration with imaging optics and associatedoptical components within a SIM system, particularly a SIM system thatis used for imaging biological samples such as nucleotide sequences.

In the context of imaging biological samples such as nucleotidesequences, SIM may provide the ability to resolve densely packedsamples, from flow cells with fluorescent signals from millions ofsample points, thereby reducing reagents needed for processing andincreasing image processing throughput. In some cases, SIM may enableresolution of fluorescent samples that are packed more densely than theAbbe diffraction limit for resolving adjoining light sources. Thebiological samples may be in regularly spaced nanowells on a flow cellor they may be in randomly distributed clusters. Adjacent nanowells maybe positioned closer together than the Abbe diffraction limit of theassociated optical system. While the present example relates tobiological samples on nanowells of a flow cell, the teachings herein maybe applied to biological samples in various other arrangements; and inother kinds of systems that employ SIM. The teachings herein are thusnot necessarily limited to imaging of biological samples.

I. Introduction

Structured illumination may produce images that have several times asmany resolved illumination sources as with normal illumination. Multipleimages with varying angles and phase displacements of structuredillumination are used to transform closely spaced, otherwiseunresolvable high spatial frequency features, into lower frequencysignals that may be sensed by an optical system without violating theAbbe diffraction limit. This limit is physically imposed on imaging bythe nature of light and optics and is expressed as a function ofemission wavelength and the numerical aperture (NA) of the finalobjective lens. Applying SIM reconstruction, information from multipleimages is transformed from the spatial domain into the Fourier domain,combined and processed, then reconstructed into an enhanced image. Theset of lower-resolution source images that are processed in a SIM systemand method may be defined as a “SIM stack.” The images in each SIM stackmay be acquired with an objective lens that is located at acorresponding z-position or distance relative to the imaged subjectmatter. Several SIM stacks may be acquired of the same subject matter,with each SIM stack having a z-position that differs from the z-positionof the other SIM stacks of the same subject matter.

In SIM, a grating is used, or an interference pattern is generated,between the illumination source and the sample, to generate anillumination pattern, such as a pattern that varies in intensityaccording to a sine or cosine function. In the SIM context, “grating” issometimes used to refer to the projected structured illuminationpattern, in addition to the surface that produces the structuredillumination pattern. The structured illumination pattern alternativelymay be generated as an interference pattern between parts of a splitcoherent beam.

Projection of structured illumination onto a sample plane, for exampleas shown in FIG. 1 , mixes the illumination pattern with fluorescent (orreflective) sources in a sample to induce a new signal, sometimes calleda Moire fringe or aliasing. The new signal shifts high-spatial frequencyinformation to a lower spatial frequency that may be captured withoutviolating the Abbe diffraction limit.

After capturing images of a sample illuminated with a 1D intensitymodulation pattern, as shown in FIG. 1A, or 2D intensity modulationpattern, as shown in FIG. 1B, a linear system of equations is solved andused to extract, from multiple images of the Moire fringe or aliasing,parts of the new signal that contains information shifted from thehigher to the lower spatial frequency.

To solve the linear equations, several images are captured with thestructured illumination pattern shifted or displaced in steps. Images ofvarying phases per angle may be captured for analysis and then separatedby bands for Fourier domain shifting and recombination. Increasing thenumber of images may improve the quality of reconstructed images byboosting the signal-to-noise ratio. However, it may also increasecomputation time. The Fourier representation of the band separatedimages is shifted and summed to produce a reconstructed sum. Eventually,an inverse Fast Fourier Transform (FFT) reconstructs a newhigh-resolution image from the reconstructed sum.

The standard algorithms for 1D modulated illumination may involvemodification when used with a 2D modulated illumination pattern. Thismay include illumination peak spacing and illumination peak angleestimation, which may involve a 2D band separation. The modification mayalso include Wicker phase estimation, which work from two points(instead of one) in order to estimate the phase in two dimensions. A 1Dinterference pattern may be generated by one dimensional diffractiongrating as shown in FIG. 1A or as a result of an interference pattern oftwo beams. In some instances, during imaging of the sample, three imagesof fringe patterns of the sample are acquired at various pattern phases(e.g., 0°, 120°, and 240°), so that each location on the sample isexposed to a range of illumination intensities, with the procedurerepeated by rotating the pattern orientation about the optical axis to 2(e.g., 45°, 135°) or 3 (e.g., 0°, 60° and 120°) separate angles.

FIG. 1B illustrates an intensity distribution that may be produced by a2D diffraction grating or by interference of two pairs of coherent lightbeams. In particular, a 2D structured illumination may be formed by twoorthogonal 1D diffraction gratings superimposed upon one another. As inthe case of 1D structured illumination patterns, the 2D illuminationpatterns may be generated either by use of 2D diffraction gratings or byinterference between two pairs of coherent light beams that creates aregularly repeating fringe pattern. Two light beams produce an intensitypattern (horizontal bright and dark lines) along y-axis and aretherefore referred to as the y-pair of incident beams. Two more lightbeams produce an intensity pattern (vertical bright and dark lines)along x-axis and are referred to as the x-pair of incident beams. Theinterference of the y-pair with the x-pair of light beams produces a 2Dillumination pattern. FIG. 1B shows intensity distribution of such a 2Dillumination pattern.

FIG. 1C illustrates an arrangement of nanowells 10 at the surface of aflow cell positioned at corners of a rectangle. FIG. 1C also shows lines20 of a structured illumination fringe pattern projected onto nanowells10. In the example shown, lines 20 are slightly angularly offsetrelative to the alignment of nanowells 10, such that lines 20 areneither perfectly aligned with (or parallel to) the rows of nanowells 10or the columns of nanowells 10. Alternatively, lines 20 may have anyother suitable spatial relationship with the alignment of columns orrows of nanowells 10; or with other spatial arrangements of nanowells10. When using 1D structured illumination, the illumination peak angleis selected such that images are taken along a line connectingdiagonally opposed corners of the rectangle. For example, two sets ofthree images (a total of six images) may be taken at +45 degree and−45-degree angles. As the distance along the diagonal is more than thedistance between any two sides of the rectangle, a higher resolutionimage is achieved. Nanowells 10 may be arranged in other geometricarrangements such as a hexagon. Three or more images may then be takenalong each of three diagonals of the hexagon, resulting, for instance,in nine or fifteen images.

II. Terminology

As used herein to refer to a structured illumination parameter, the term“frequency” is intended to refer to an inverse of spacing betweenfringes or lines of a structured illumination pattern (e.g., fringe orgrid pattern), as frequency and period are inversely related. Forexample, a pattern having a greater spacing between fringes will have alower frequency than a pattern having a lower spacing between fringes.

As used herein to refer to a structured illumination parameter, the term“phase” is intended to refer to a phase of a structured illuminationpattern illuminating a sample. For example, a phase may be changed bytranslating a structured illumination pattern relative to an illuminatedsample.

As used herein to refer to a structured illumination parameter, the term“orientation” is intended to refer to a relative orientation between astructured illumination pattern (e.g., fringe or grid pattern) and asample illuminated by the pattern. For example, an orientation may bechanged by rotating a structured illumination pattern relative to anilluminated sample.

As used herein to refer to a structured illumination parameter, theterms “predict” or “predicting” are intended to mean either (i)calculating the value(s) of the parameter without directly measuring theparameter or (ii) estimating the parameter from a captured imagecorresponding to the parameter. For example, a phase of a structuredillumination pattern may be predicted at a time t1 by interpolationbetween phase values directly measured or estimated (e.g., from capturedphase images) at times t2 and t3 where t2<t1<t3. As another example, afrequency of a structured illumination pattern may be predicted at atime t1 by extrapolation from frequency values directly measured orestimated (e.g., from captured phase images) at times t2 and t3 wheret2<t3<t1.

As used herein to refer to light diffracted by a diffraction grating,the term “order” or “order number” is intended to mean the number ofinteger wavelengths that represents the path length difference of lightfrom adjacent slits or structures of the diffraction grating forconstructive interference. The interaction of an incident light beam ona repeating series of grating structures or other beam splittingstructures may redirect or diffract portions of the light beam intopredictable angular directions from the original beam. The term “zerothorder” or “zeroth order maximum” is intended to refer to the centralbright fringe emitted by a diffraction grating in which there is nodiffraction. The term “first-order” is intended to refer to the twobright fringes diffracted to either side of the zeroth order fringe,where the path length difference is ±1 wavelengths. Higher orders arediffracted into larger angles from the original beam. The properties ofthe grating may be manipulated to control how much of the beam intensityis directed into various orders. For example, a phase grating may befabricated to maximize the transmission of the non-zeroth orders andminimize the transmission of the zeroth order beam.

As used herein, the term “optical transfer function” or, in itsabbreviated form “OTF,” is intended to mean the complex valued transferfunction describing an imaging system's response as a function of thespatial frequency. The OTF may be derived from the Fourier transform ofthe point spread function. In examples described herein, only theamplitude portion of the OTF is important. The amplitude portion of theOTF may be referred to as the “modulation transfer function” or, in itsabbreviated form, the “MTF.”

As used herein to refer to a sample, the term “feature” is intended tomean a point or area in a pattern that may be distinguished from otherpoints or areas according to relative location. An individual featuremay include one or more molecules of a particular type. For example, afeature may include a single target nucleic acid molecule having aparticular sequence or a feature may include several nucleic acidmolecules having the same sequence (and/or complementary sequence,thereof).

As used herein, the term “xy plane” is intended to mean a 2-dimensionalarea defined by straight line axes x and y in a Cartesian coordinatesystem. When used in reference to a detector and an object observed bythe detector, the area may be further specified as being orthogonal tothe beam axis, or the direction of observation between the detector andobject being detected.

As used herein, the term “z coordinate” is intended to mean informationthat specifies the location of a point, line or area along an axis thatis orthogonal to an xy plane in a Cartesian coordinate system. Inparticular implementations, the z axis is orthogonal to an area of anobject that is observed by a detector. For example, the direction offocus for an optical system may be specified along the z axis.

As used herein, the term “optically coupled” is intended to refer to oneelement being adapted to impart light to another element directly orindirectly.

As used herein, an element or step recited in the singular and proceededwith the word “a” or “an” should be understood as not excluding pluralof said elements or steps, unless such exclusion is explicitly stated.Furthermore, references to “one implementation” are not intended to beinterpreted as excluding the existence of additional implementationsthat also incorporate the recited features. Moreover, unless explicitlystated to the contrary, implementations “comprising” or “having” anelement or a plurality of elements having a particular property mayinclude additional elements whether or not they have that property.

The terms “substantially,” “about,” and “approximately” used throughoutthis Specification are used to describe and account for smallfluctuations, such as due to variations in processing. For example, theymay refer to less than or equal to ±5%, such as less than or equal to±2%, such as less than or equal to ±1%, such as less than or equal to±0.5%, such as less than or equal to ±0.2%, such as less than or equalto ±0.1%, such as less than or equal to ±0.05%.

The term “based on” should be understood to mean that something isdetermined at least in part by the thing it is indicated as being “basedon.” To indicate that something must necessarily be completelydetermined by something else, it is described as being based exclusivelyon whatever it is completely determined by.

As used herein, the term “nucleotide sequence” or “polynucleotidesequence” should be read to include a polynucleotide molecule, as wellas the underlying sequence of the molecule, depending on context. Asequence of a polynucleotide may contain (or encode) informationindicative of certain physical characteristics.

III. Examples of Imaging System Components and Arrangements

In some implementations of SIM systems, a linearly polarized light beamis directed through an optical beam splitter that splits the beam intotwo or more separate orders that may be combined and projected on theimaged sample as an interference fringe pattern with a sinusoidalintensity variation. The split beams are equivalent in power in order toachieve maximum modulation at the sample plane. Diffraction gratings areexamples of beam splitters that may generate beams with a high degree ofcoherence and stable propagation angles. When two such beams arecombined, the interference between them may create a uniform,regularly-repeating fringe pattern where the spacing is determined byfactors including the angle between the interfering beams. Therelationship between the fringe periodicity (FP), the incidence angle(θ) and the wavelength of light (λ) may be expressed in the followingequation (I):

FP=λ÷2 sin(θ),  (I)

where the fringe period (FP) and the wavelength of light (λ) are in thesame units (e.g., nm) and θ is the incidence angle with respect to thesurface normal expressed in radians.

FIGS. 2-4B illustrate examples of different forms that SIM imagingsystems may take. It should be noted that while these systems aredescribed primarily in the context of SIM imaging systems that generate1D illumination patterns, the technology disclosed herein may beimplemented with SIM imaging systems that generate higher dimensionalillumination patterns (e.g., two-dimensional grid patterns).

FIG. 2 illustrates a SIM imaging system 100 that may implementstructured illumination parameter prediction in accordance with someimplementations described herein. For example, system 100 may be astructured illumination fluorescence microscopy system that utilizesspatially structured excitation light to image a biological sample.

In the example of FIG. 2 , a light emitter 150 is configured to output alight beam that is collimated by collimation lens 151. The collimatedlight is structured (patterned) by light structuring optical assembly155 and directed by dichroic mirror 160 through objective lens 142 ontoa sample of a sample container 110, which is positioned on a motionstage 170. In the case of a fluorescent sample, the sample fluoresces inresponse to the structured excitation light, and the resultant light iscollected by objective lens 142 and directed to an image sensor ofcamera system 140 to detect fluorescence.

Light structuring optical assembly 155 includes one or more opticaldiffraction gratings or other beam splitting elements (e.g., a beamsplitter cube or plate) to generate a pattern of light (e.g., fringes,typically sinusoidal) that is projected onto samples of a samplecontainer 110. The diffraction gratings may be one-dimensional ortwo-dimensional transmissive or reflective gratings. The diffractiongratings may be sinusoidal amplitude gratings or sinusoidal phasegratings. In some versions, light structuring optical assembly 155includes a pair of phase masks, where each phase mask includes a pieceof glass with graduations etched into the glass.

In some implementations, the diffraction grating(s)s may not utilize arotation stage to change an orientation of a structured illuminationpattern. In other implementations, the diffraction grating(s) may bemounted on a rotation stage. In some implementations, the diffractiongratings may be fixed during operation of the imaging system (i.e., notrequire rotational or linear motion). For example, in a particularimplementation, further described below, the diffraction gratings mayinclude two fixed one-dimensional transmissive diffraction gratingsoriented perpendicular to each other (e.g., a horizontal diffractiongrating and vertical diffraction grating).

As illustrated in the example of FIG. 2 , light structuring opticalassembly 155 outputs the first orders of the diffracted light beamswhile blocking or minimizing all other orders, including the zerothorders. However, in alternative implementations, additional orders oflight may be projected onto the sample.

During each imaging cycle, imaging system 100 utilizes light structuringoptical assembly 155 to acquire a plurality of images at various phases,with the fringe pattern displaced laterally in the modulation direction(e.g., in the x-y plane and perpendicular to the fringes), with thisprocedure repeated one or more times by rotating the pattern orientationabout the optical axis (i.e., with respect to the x-y plane of thesample). The captured images may then be computationally reconstructedto generate a higher resolution image (e.g., an image having about twicethe lateral spatial resolution of individual images).

In system 100, light emitter 150 may be an incoherent light emitter(e.g., emit light beams output by one or more excitation diodes), or acoherent light emitter such as emitter of light output by one or morelasers or laser diodes. As illustrated in the example of system 100,light emitter 150 includes an optical fiber 152 for guiding an opticalbeam to be output. However, other configurations of a light emitter 150may be used. In implementations utilizing structured illumination in amulti-channel imaging system (e.g., a multi-channel fluorescencemicroscope utilizing multiple wavelengths of light), optical fiber 152may optically couple to a plurality of different light sources (notshown), each light source emitting light of a different wavelength.Although system 100 is illustrated as having a single light emitter 150,in some implementations multiple light emitters 150 may be included. Forexample, multiple light emitters may be included in the case of astructured illumination imaging system that utilizes multiple arms,further discussed below.

In some implementations, system 100 may include a projection lens 156that may include a lens element to articulate along the z-axis to adjustthe structured beam shape and path. For example, a component of theprojection lens 156 may be articulated to account for a range of samplethicknesses (e.g., different cover glass thickness) of the sample incontainer 110.

In the example of system 100, fluid delivery module or device 190 maydirect the flow of reagents (e.g., fluorescently labeled nucleotides,buffers, enzymes, cleavage reagents, etc.) to (and through) samplecontainer 110 and waste valve 120. Sample container 110 may include oneor more substrates upon which the samples are provided. For example, inthe case of a system to analyze a large number of different nucleic acidsequences, sample container 110 may include one or more substrates onwhich nucleic acids to be sequenced are bound, attached or associated.The substrate may include any inert substrate or matrix to which nucleicacids may be attached, such as for example glass surfaces, plasticsurfaces, latex, dextran, polystyrene surfaces, polypropylene surfaces,polyacrylamide gels, gold surfaces, and silicon wafers. In someapplications, the substrate is within a channel or other area at aplurality of locations formed in a matrix or array across the samplecontainer 110. System 100 may also include a temperature stationactuator 130 and heater/cooler 135 that may optionally regulate thetemperature of conditions of the fluids within the sample container 110.

In particular implementations, the sample container 110 may beimplemented as a patterned flow cell including a transparent coverplate, a substrate, and a liquid contained therebetween, and abiological sample may be located at an inside surface of the transparentcover plate or an inside surface of the substrate. The flow cell mayinclude a large number (e.g., thousands, millions, or billions) of wells(also referred to as nanowells) or regions that are patterned into adefined array (e.g., a hexagonal array, rectangular array, etc.) intothe substrate. Each region may form a cluster (e.g., a monoclonalcluster) of a biological sample such as DNA, RNA, or another genomicmaterial which may be sequenced, for example, using sequencing bysynthesis. The flow cell may be further divided into a number of spacedapart lanes (e.g., eight lanes), each lane including a hexagonal arrayof clusters.

Sample container 110 may be mounted on a sample stage 170 to providemovement and alignment of the sample container 110 relative to theobjective lens 142. The sample stage may have one or more actuators toallow it to move in any of three dimensions. For example, in terms ofthe Cartesian coordinate system, actuators may be provided to allow thestage to move in the x, y, and z directions relative to the objectivelens. This may allow one or more sample locations on sample container110 to be positioned in optical alignment with objective lens 142.Movement of sample stage 170 relative to objective lens 142 may beachieved by moving the sample stage itself, the objective lens, someother component of the imaging system, or any combination of theforegoing. Further implementations may also include moving the entireimaging system over a stationary sample. Alternatively, sample container110 may be fixed during imaging.

In some implementations, a focus (z-axis) component 175 may be includedto control positioning of the optical components relative to the samplecontainer 110 in the focus direction (typically referred to as the zaxis, or z direction). Focus component 175 may include one or moreactuators physically coupled to the optical stage or the sample stage,or both, to move sample container 110 on sample stage 170 relative tothe optical components (e.g., the objective lens 142) to provide properfocusing for the imaging operation. For example, the actuator may bephysically coupled to the respective stage such as, for example, bymechanical, magnetic, fluidic or other attachment or contact directly orindirectly to or with the stage. The one or more actuators may beconfigured to move the stage in the z-direction while maintaining thesample stage in the same plane (e.g., maintaining a level or horizontalattitude, perpendicular to the optical axis). The one or more actuatorsmay also be configured to tilt the stage. This may be done, for example,so that sample container 110 may be leveled dynamically to account forany slope in its surfaces.

The structured light emanating from a test sample at a sample locationbeing imaged may be directed through dichroic mirror 160 to one or moredetectors of camera system 140. In some implementations, a filterswitching assembly 165 with one or more emission filters may beincluded, where the one or more emission filters may be used to passthrough particular emission wavelengths and block (or reflect) otheremission wavelengths. For example, the one or more emission filters maybe used to switch between different channels of the imaging system. In aparticular implementation, the emission filters may be implemented asdichroic mirrors that direct emission light of different wavelengths todifferent image sensors of camera system 140.

Camera system 140 may include one or more image sensors to monitor andtrack the imaging (e.g., sequencing) of sample container 110. Camerasystem 140 may be implemented, for example, as a charge-coupled device(CCD) image sensor camera, but other image sensor technologies (e.g.,active pixel sensor) may be used. While camera system 140 and associatedoptical components are shown as being positioned above sample container110 in FIG. 2 , one or more image sensors or other camera components maybe incorporated into system 100 in numerous other ways as will beapparent to those skilled in the art in view of the teachings herein.For instance, one or more image sensors may be positioned under samplecontainer 110 or may even be integrated into sample container 110.

Output data (e.g., images) from camera system 140 may be communicated toa real-time SIM imaging component 191 that may be implemented as asoftware application that, as further described below, may reconstructthe images captured during each imaging cycle to create an image havinga higher spatial resolution. The reconstructed images may take intoaccount changes in structure illumination parameters that are predictedover time. In addition, SIM imaging component 191 may be used to trackpredicted SIM parameters and/or make predictions of SIM parameters givenprior estimated and/or predicted SIM parameters.

A controller 195 may be provided to control the operation of structuredillumination imaging system 100, including synchronizing the variousoptical components of system 100. The controller may be implemented tocontrol aspects of system operation such as, for example, configurationof light structuring optical assembly 155 (e.g., selection and/or lineartranslation of diffraction gratings), movement of projection lens 156,activation of focus component 175, stage movement, and imagingoperations. The controller may be also be implemented to controlhardware elements of the system 100 to correct for changes in structuredillumination parameters over time. For example, the controller may beconfigured to transmit control signals to motors or other devicescontrolling a configuration of light structuring optical assembly 155,motion stage 170, or some other element of system 100 to correct orcompensate for changes in structured illumination phase, frequency,and/or orientation over time. In implementations, these signals may betransmitted in accordance with structured illumination parameterspredicted using SIM imaging component 191. In some implementations,controller 195 may include a memory for storing predicted and orestimated structured illumination parameters corresponding to differenttimes and/or sample positions.

In various implementations, the controller 195 may be implemented usinghardware, algorithms (e.g., machine executable instructions), or acombination of the foregoing. For example, in some implementations thecontroller may include one or more CPUs, GPUs, or processors withassociated memory. As another example, the controller may comprisehardware or other circuitry to control the operation, such as a computerprocessor and a non-transitory computer readable medium withmachine-readable instructions stored thereon. For example, thiscircuitry may include one or more of the following: field programmablegate array (FPGA), application specific integrated circuit (ASIC),programmable logic device (PLD), complex programmable logic device(CPLD), a programmable logic array (PLA), programmable array logic (PAL)and other similar processing device or circuitry. As yet anotherexample, the controller may comprise a combination of this circuitrywith one or more processors.

FIG. 3 shows an example of an alternative optical assembly 200 that maybe incorporated into system (e.g., in place of optical assembly 155).Optical assembly 200 of this example includes a light emitting assembly210, a fixed reflecting element 220, a phase mask assembly 230, agrating switcher 250, an adjustable reflecting element 270, and aprojection lens assembly 280. Light emitting assembly 210 may includevarious components, including but not limited to a source of coherentlight (e.g., at least one laser, etc.) and a pair of anamorphic prisms,a source of incoherent light and a collimator, or any other suitablecomponents as will be apparent to those skilled in the art in view ofthe teachings herein. In some versions light emitting assembly 210 isoperable to emit light via two or more separate channels (e.g., a bluechannel and a green channel). In versions where light is emitted in twoor more separate channels, system 100 may include two or morecorresponding image sensors, such that each image sensor is dedicated toa corresponding image sensor. Also, in some versions, light emittingassembly 210 is operable to emit light in pulses at a predeterminedfrequency (e.g., using a high-speed shutter, etc.).

Reflecting element 220 of the present example includes a mirror whoseposition is fixed relative to the other components of optical assembly200. As described in greater detail below, reflecting element 220 ispositioned and configured to reflect light emitted from light emittingassembly 210 toward phase mask assembly 230 and grating switcher 250during operation of optical assembly 200.

As best seen in FIG. 4 , phase mask assembly 230 of the present exampleincludes a pair of triangular glass elements 232, 242 fixedly mounted toa base 240. Each glass element 232, 242 includes a reflector 234, 244along one side of the glass element 232, 242. Each glass element 232,242 also includes a phase mask 236, 246 along another side of the glasselement 232, 242. In the present example, each phase mask 236, 246includes graduations (e.g., parallel slits or grooves, etc.) forming agrating or fringe pattern etched into the glass of glass element 232,242. The graduation spacing may be chosen to diffract light at suitableangles and tuned to the minimum resolvable feature size of the imagedsamples for operation of system 100. As will be described in greaterdetail below, these phase masks 236, 246 are configured to produce Moirefringe or aliasing during operation of optical assembly 200. While phasemasks 236, 246 are formed by etched graduations in the glass of glasselements 232, 242 in the present example, other suitable ways in whichphase masks 236, 246 may be formed will be apparent to those skilled inthe art in view of the teachings herein. During operation of opticalassembly 200, the entire phase mask assembly 230 remains stationaryrelative to the other components of optical assembly 200.

To improve efficiency of the system, the zeroth order beams and allother higher order diffraction beams output by each phase mask 236, 246may be blocked (i.e., filtered out of the illumination pattern projectedon the sample 110). For example, a beam blocking element (not shown)such as an order filter may be inserted into the optical after pathphase mask assembly 230. In some implementations, diffraction gratingsphase masks 236, 246 may configured to diffract the beams into only thefirst orders and the zeroth order (undiffracted beam) may be blocked bysome beam blocking element.

As shown in FIG. 3 , grating switcher 250 of the present exampleincludes a plate 252 mounted to a shaft 254. Shaft 254 is furthercoupled with a motor 256 that is operable to rotate shaft 254 and plate252 about an axis A. One end 260 of plate 252 includes a pair of mirrors262, 264 with each mirror 262, 264 being mounted to an opposite side ofplate 252. The other end 266 of plate 252 defines an opening 268 thatallows light to pass through as described below. In some versions, motor256 is a stepper motor. Alternatively, motor 256 may take any othersuitable form; and motor 256 may be substituted with any other suitablesource of rotary motion. As shown in FIGS. 5A-5D and as will bedescribed in greater detail below, motor 256 may be activated totransition grating switcher 250 between a first state (FIGS. 5A-5B) anda second state (FIGS. 5C-5D) by rotating shaft 254 and plate 252 aboutthe axis A. When grating switcher 250 is in the first state, gratingswitcher 250 and phase mask assembly 230 may provide a first gratingangle. When grating switcher 250 is in the second state, gratingswitcher 250 and phase mask assembly 230 may provide a second gratingangle.

As also shown in FIG. 3 , adjustable reflecting element 270 of thepresent example includes a mirror that is coupled with an actuator 272,such that the actuator 272 is operable to drive reflecting element 270along a linear path LP1. In this example, linear path LP1 is parallelwith axis A. In some versions, actuator 272 includes a piezoelectricelement. As another example, actuator 272 may include a solenoid. Insome other versions, actuator 272 includes a stepper motor or otherrotary drive source that is coupled with a mechanical assembly (e.g.,rack and pinion or worm gear and nut, etc.) that is operable to convertrotary motion into linear motion. As described in greater detail below,with actuator 272 changing the position of reflecting element 270 alonglinear path LP1, actuator 272 and reflecting element 270 are togetheroperable to provide phase modulation to light that is transmittedthrough optical assembly 200. In other words, actuator 272 andreflecting element 270 may together provide a phase adjustment assembly.

By way of example, actuator 272 may be operable to drive reflectingelement 270 through a range of motion of approximately 5 μm duringoperation of actuator 272, which may provide fringe movement ofapproximately 240 degrees, as described in greater detail below.Alternatively, actuator 272 may be operable to drive reflecting element270 through a range of motion ranging from approximately 2 μm toapproximately 10 μm during operation of actuator 272. As described ingreater detail below, actuator 272 may be driven to arrest motion ofreflecting element at two, three, or more different positions throughthe range of motion along the linear path.

Projection lens assembly 280 may include one or more lens elements(e.g., a tube lens) and various other components as will be apparent tothose skilled in the art in view of the teachings herein. Light passedthrough projection lens assembly 280 may eventually reach samplecontainer 110 (e.g., a flow cell, etc.). In some instances, this maycause biological material in the sample container 110 to fluoresce, withsuch fluorescence being picked up by an image sensor (e.g., an imagesensor of camera system 140) to enable analysis of the biologicalmaterial. Projection lens assembly 280 of the present example is coupledwith an actuator 282, which is operable to drive at least a portion ofprojection lens assembly 280 along a linear path LP2. In some versions,actuator 282 includes a piezoelectric element. As another example,actuator 282 may include a solenoid. In some other versions, actuator282 includes a stepper motor or other rotary drive source that iscoupled with a mechanical assembly (e.g., rack and pinion or worm gearand nut, etc.) that is operable to convert rotary motion into linearmotion. As described in greater detail below, with actuator 282 changingthe position of at least a portion of projection lens assembly 280 alonglinear path LP2, actuator 282 and projection lens assembly 280 aretogether operable to provide adjustment of the SIM grating focal plane.

As noted above, system 100 of the present example includes a controller195. Controller 195 may be used to control the operation of opticalassembly 200 and other features of system 100, including synchronizingthe various components of optical assembly 200 and system 100. Thecontroller 195 may be implemented to control aspects of system operationsuch as, for example, activation of motor 256, activation of actuator272, movement of one or more elements of projection lens assembly 280via actuator 282, activation of focus component 175, activation ofcamera system 140, and other imaging operations. The controller may bealso be implemented to control hardware elements of the system 100 tocorrect for changes in structured illumination parameters over time. Forexample, the controller may be configured to transmit control signals todevices (e.g., motor 256, actuator 272, etc.) to correct or compensatefor changes in structured illumination phase, frequency, and/ororientation over time. In implementations, these signals may betransmitted in accordance with structured illumination parameterspredicted using a SIM imaging component. In some implementations, thecontroller may include a memory for storing predicted and or estimatedstructured illumination parameters corresponding to different timesand/or sample positions.

FIGS. 5A-5D show optical assembly 200 at various stages of operation. Atthe stage shown in FIG. 5A, light emitting assembly 210 emits lighttoward reflecting element 220, which reflects the light toward phasemask assembly 230 and grating switcher 250. At this stage, gratingswitcher 250 is in a first state such that the light reflected fromreflecting element 220 is further reflected by mirror 262. The lightreflected by mirror 262 passes through glass element 242 and reachesreflector 244, which reflects the light toward phase mask 246. As thelight passes through phase mask 246, phase mask 246 provides a patternedform to the light. This patterned or structured light then passesthrough opening 268 of plate 252 and reaches reflecting element 270,which then reflects the structured light toward projection lens assembly280. After passing through projection lens assembly 280, the structuredlight reaches the object targeted for imaging (e.g., the samplecontainer 110); and camera system 140 captures a first image of thetargeted object.

After the first image is acquired with the configuration of opticalsystem 200 shown in FIG. 5A, actuator 272 is activated to drivereflecting element 270 from a first position on the linear path LP1 to asecond position on the linear path LP1, such that optical system 200 isthen in the configuration shown in FIG. 5B. At the stage shown in FIG.5B, light emitting assembly 210 emits light toward reflecting element220, which reflects the light toward phase mask assembly 230 and gratingswitcher 250. At this stage, grating switcher 250 is in a first statesuch that the light reflected from reflecting element 220 is furtherreflected by mirror 262. The light reflected by mirror 262 passesthrough glass element 242 and reaches reflector 244, which reflects thelight toward phase mask 246. As the light passes through phase mask 246,phase mask 246 provides a patterned form to the light. This patterned orstructured light then passes through opening 268 of plate 252 andreaches reflecting element 270, which then reflects the structured lighttoward projection lens assembly 280. After passing through projectionlens assembly 280, the structured light reaches the object targeted forimaging (e.g., the sample container 110); and camera system 140 capturesanother image of the targeted object.

The only difference between the stage shown in FIG. 5A and the stageshown in FIG. 5B is that reflecting element 270 is in a second state(i.e., at a second position along the linear path LP1). Thus, becausereflecting element 270 is at a different position during this stage ofoperation, the image captured with optical assembly 200 in theconfiguration shown in FIG. 5B will have a different phase than theimage captured with optical assembly 200 in the configuration shown inFIG. 5A.

In some versions of the process described herein, actuator 272 isactivated to drive reflecting element 270 to a third position alonglinear path LP1 while grating switcher 250 is in the first state, beforeproceeding to the stage shown in FIG. 5C and described below. In suchversions of the process, camera system 140 may capture three imageswhile grating switcher 250 is in the first state, with each of theseimages representing a different phase based on the respective positionsof reflecting element 270 along the linear path LP1. Of course, actuator272 may also be activated to drive reflecting member 270 to a fourthposition, fifth position, etc., such that any desired number of phasesmay be employed during the capture of images while grating switcher 250is in the first state.

After the desired number of images have been acquired with gratingswitcher 250 in the first state shown in FIGS. 5A-5B, motor 256 isactivated to rotate shaft 254 about the axis A, thereby rotating plate252 about the axis A, to transition grating switcher 250 to the secondstate shown in FIGS. 5C-5D. At the stage shown in FIG. actuator 272 hasalso been activated to return reflecting element 270 from the secondstate (i.e., the second position on the linear path LP1) back to thefirst state (i.e., the first position on the linear path LP1). In someother versions, reflecting element 270 remains in the second stateimmediately following the transition of grating switcher 250 from thefirst state to the second state; and reflecting element 270 istransitioned to the first state after an image has been captured whilereflecting element 270 is in the second state and grating switcher 250is in the second state.

At the stage shown in FIG. 5C, light emitting assembly 210 emits lighttoward reflecting element 220, which reflects the light toward phasemask assembly 230 and grating switcher 250. With grating switcher 250now in the second state, the light reflected from reflecting element 220passes through opening 268 and passes further through glass element 232.The light passed through glass element 232 reaches reflector 234, whichreflects the light toward phase mask 236. As the light passes throughphase mask 236, phase mask 236 provides a patterned form to the light.This patterned or structured light is then reflected off of mirror 264.Mirror 264 reflects the structured light toward reflecting element 270,which then reflects the structured light toward projection lens assembly280. After passing through projection lens assembly 280, the structuredlight reaches the object targeted for imaging (e.g., the samplecontainer 110); and camera system 140 captures another image of thetargeted object.

After the image is acquired with the configuration of optical system 200shown in FIG. 5C, actuator 272 is activated to drive reflecting element270 from the first state (i.e., the first position on the linear pathLP1) to the second state (i.e., the second position on the linear pathLP1), such that optical system 200 is then in the configuration shown inFIG. 5D. At the stage shown in FIG. 5D, light emitting assembly 210emits light toward reflecting element 220, which reflects the lighttoward phase mask assembly 230 and grating switcher 250. With gratingswitcher 250 now in the second state, the light reflected fromreflecting element 220 passes through opening 268 and passes furtherthrough glass element 232. The light passed through glass element 232reaches reflector 234, which reflects the light toward phase mask 236.As the light passes through phase mask 236, phase mask 236 provides apatterned form to the light. This patterned or structured light is thenreflected off of mirror 264. Mirror 264 reflects the structured lighttoward reflecting element 270, which then reflects the structured lighttoward projection lens assembly 280. After passing through projectionlens assembly 280, the structured light reaches the object targeted forimaging (e.g., the sample container 110); and camera system 140 capturesanother image of the targeted object.

The only difference between the stage shown in FIG. 5C and the stageshown in FIG. 5D is that reflecting element 270 is in the second state(i.e., at the second position along the linear path LP1). Thus, becausereflecting element 270 is at a different position during this stage ofoperation, the image captured with optical assembly 200 in theconfiguration shown in FIG. 5D will have a different phase than theimage captured with optical assembly 200 in the configuration shown inFIG. 5C.

In some versions of the process described herein, actuator 272 isactivated to drive reflecting element 270 to a third position alonglinear path LP1 while grating switcher 250 is in the second state,before completing the process of capturing images. In such versions ofthe process, camera system 140 may capture three images while gratingswitcher 250 is in the second state, with each of these imagesrepresenting a different phase based on the respective positions ofreflecting element 270 along linear path LP1. Of course, actuator 272may also be activated to drive reflecting member 270 to a fourthposition, fifth position, etc., such that any desired number of phasesmay be employed during the capture of images while grating switcher 250is in the second state.

As noted above, the image capture process may be carried out through twoor more separate channels (e.g., a blue channel and a green channel). Inother words, the process described above with reference to FIGS. 5A-5Dmay be carried out through two or more separate channels. Light emittingassembly 210 may be operable to provide both channels; or each channelmay have its own light emitting assembly 210. In some versions, the twoseparate channels are activated simultaneously through optical assembly200. In some other versions, a first channel is activated during thestage shown in FIG. 5A, then a second channel is activated during thestage shown in FIG. 5A, then the first channel is activated during thestage shown in FIG. 5B, then the second channel is activated during thestage shown in FIG. 5B, and so on, until the second channel is activatedduring the stage shown in FIG. 5D. As yet another example, each channelmay have its own dedicated optical assembly 200. In some such versions,further optical components may be utilized to enable the projection lensassembly 280 of each optical assembly 200 to project the light from eachchannel to the same target (e.g., sample container 110). Other suitableways in which one or more optical assemblies 200 may enable use of twoor more channels will be apparent to those skilled in the art in view ofthe teachings herein. It should also be understood that other componentswithin system 100 (e.g., filter switching assembly 165) may furtherenable use of two or more channels. In versions where one channel isblue and another channel is green, the blue channel may operate withlight at a wavelength in the range from approximately 450 nm toapproximately 500 nm; and the green channel may operate with light at awavelength in the range from approximately 500 nm to approximately 570nm.

As also noted above, the subject matter that is imaged with use ofoptical assembly 200 in system 100 may include one or more biologicalsamples (e.g., nucleotides, etc.) in nanowells on a flow cell, such thatsome forms of sample container 110 may include flow cell. Such nanowellsmay be arranged in a regular repeating pattern. For a rectangularpattern, two structured illumination angles may be used, substantiallyalong two diagonals connecting opposing corners of a rectangle in thepattern, so that intensity peaks of the structured illumination areoriented substantial normal to the two diagonals. Alternatively, thestructured illumination angle may be oriented along the same directionas the rectangular nanowell pattern direction (i.e., not along theopposing corners of the rectangle).

For a repeating hexagonal pattern of nanowells, with three diagonalsconnecting opposing corners of hexagons in the pattern, three structuredillumination angles may be used with intensity peaks that are orientedsubstantial normal to the three diagonals. Alternatively, a two-angleillumination pattern may be used in conjunction with a flow cell havinga hexagonal pattern of nanowells, such that it is not necessary in allcases to use three structured illumination angles in conjunction with ahexagonal pattern of nanowells. Moreover, the structured illuminationangle may be oriented along the same direction as the hexagonal nanowellpattern direction (i.e., not along the opposing corners of the hexagon).

Regardless of the kind of pattern of nanowells, adjacent nanowells maybe positioned closer together than the Abbe diffraction limit of theassociated optical system. Alternatively, samples may be randomlydistributed over an imaging plane without nanowells. Or, the samples maybe regularly arranged over the imaging plane on some structure otherthan nanowells.

IV. Examples of Image Processing Algorithms

A. Overview of SIM Image Processing Method

An image captured by an optical sensor or image sensor (e.g., asintegrated into camera system 140) may be referred to as a tile. Imageprocessing algorithms as described below may subdivide a captured imagetile into sub-tiles. Each sub-tile may be evaluated independently. Anear-center sub-tile may be handled differently than other sub-tiles. Animaging cycle for a flow cell may capture many image tiles with someoverlap. Sub-tiles may be reconstructed independently of one another,even in parallel. Reconstructions from enhanced sub-tiles may bestitched together to create a reconstructed tile with enhanced spatialresolution. In some instances, an image tile is subdivided intosub-tiles such that the peak lines are approximately evenly spacedwithin a sub-tile, thereby achieving better image quality fromreconstructed sub-tiles across a field of view of a lens.

In some instances, at least three parameters are mapped for eachsub-tile. Such parameters may include illumination peak angle,illumination peak spacing, and phase displacement. The illumination peakangle may also be referred to as grating angle. The illumination peakspacing may also be referred to as grating spacing. In other words, theillumination peak spacing defines the periodicity of the grating (e.g.,the spacing between parallel lines defined by phase masks 236, 246). Thephase displacement or the phase is the shift of the structuredillumination pattern or grating as projected onto the sample plane(e.g., based on the position of reflecting element 270 along the linearpath LP1, as driven by actuator 272). In other words, the phase may bedefined as the distance from a common reference point to the start ofthe repeating illumination pattern in the direction orthogonal to thegrating. The phase may be expressed in radians or degrees; and may beregarded as a fraction of the repeating pattern periodicity. The phasedisplacement may also be referred to as the grating phase. The angle andspacing may be mapped using quadratic surface distortion models.

The following describes examples of techniques that may be used toestimate parameters for SIM image reconstruction. Some of the techniquesdisclosed compensate for fringe peak lines that are distorted or bentdue lens imperfections. Pattern lines that are supposed to be parallelbegin that way near the center of the image but tend to converge orbecome non-parallel near the edge of the lens. This impacts illuminationpeak angle or orientation, illumination peak spacing, and phase offset.FIG. 8A illustrates dividing an image tile into overlapping regionsreferred to as sub-tiles or sub-windows or sub-fields. The sub-tiles aresmall enough that parameters may be set that will give satisfactoryreconstruction for a whole sub-tile. In some versions, each sub-tileincludes 512 by 512 pixels of the optical sensor. Larger or smallernumbers may be used, including but not limited to 256, 400, 1024, 2048and 4096; or in a range from 256 to 4096 pixels. The sub-tiles mayoverlap by at least 2 pixels of the optical sensor. Larger or smallernumbers may be used. For example, for a 512-pixel wide window, up to a256-pixel overlap may be used; and for 1024 pixels wide, up to a 512overlap may be used.

The parameter estimation may be performed in two steps. First, parameterestimation may be performed for a near-center sub-tile of the image.Then, parameter estimation may be performed for other sub-tiles andcompared to the near-center sub-tile to determine distortions andcorrections for the distortions, relative to parameters for thenear-center sub-tile.

FIGS. 6A to 6C illustrate physical aspects of the full field of view(FOV). In one implementation, the rectangular sensor is used that is5472 pixels by 3694 pixels. Of course, a square sensor or a differentsize of sensor may be used, for example, 5472×5472 pixels, or 4800×4800pixels. When a rectangular sensor is used, distortion is greatestclosest to the edge of the lens. A lens often is round, so a rectangularsensor does not come as close to the edge of the lens on the long sideas it does on the short side.

FIG. 6A presents two illustrations that show fringe spacing distortionacross the full field of view (FOV). The FIG. 300 on the left is asimplified depiction 300 of bending parallel lines due to distortion ofa lens that magnifies. The lines depicted are intended to be parallel inthe image plane. Viewed through a lens, they appear to converge at rightand left ends, relative to spacing in the center. The FIG. 302 on theright is another exaggerated example. In this figure the fringe linesare oriented diagonally between top left and bottom right corners. Thefringe spacing is exaggerated to make it easier to see. The fringe linesconverge at the top left and bottom right corners, relative to thecenter. For a particular manufacturer's lens, the fringe pattern may benon-uniform.

FIGS. 6B and 6C depict measurements of spacing in an image betweennominally parallel fringe peaks in the image plane, for green and bluelaser illumination. The color scale indicates a variation in spacingbetween 2.8 and 2.22. In both drawings, the color scale indicates thatthe center spacing between parallel lines is approximately 2.14.Irregularity under green wavelength illumination is seen in the topright-hand corner of FIG. 6B. More substantial irregularity under bluewavelength illumination is seen in FIG. 6C, along the right and leftedges. In these figures, the fringe pattern was a series of parallellines at an angle of 45°, from bottom left to top right of the figures.Thus, the spacing is measured in the direction of the arrow in FIG. 8C.These figures motivate correction of distortions caused by the lens.Since lenses are individually manufactured and mounted, calibration andcorrection of individual systems after assembly is desirable.

FIG. 6D illustrates sub-tiles or subfields of the full field of view(FOV) in an image tile. In this figure, the sub-tile illustrated is 512pixels by 512 pixels. These sub-tiles may subdivide the field of vision,shown, or may overlap. Sub-tiles may be larger or smaller. For instance,400×400 and 1024×1024 pixel sub-tiles have been shown to be workable.The figure illustrates 5×7 sub-tiles. The larger sensor called out abovemay have 8×11 sub-tiles. Other configurations of sub-tiles such as 3×3,5×5, 5×7, 9×9, 9×16 may be used. Larger sensors may be divided into moresub-tiles. The sub-tiles may overlap by at least 2 pixels of the opticalsensor. Larger and smaller number of pixels may be used for overlappingbetween sub-tiles. For example, for a 512-pixel wide sub-tile, up to a256-pixel overlap may be used, and for a 1024-pixel wide sub-tile, up toa 256-pixel overlap may be used. Consistent with FIGS. 6B and 6C, thereare several candidate near-center sub-tiles 304, all in the sweet spotof the lens, including a center sub-tile in an odd×odd sub-tile array.As used herein, a near-center sub-tile either includes a center pixel ofthe sensor or abuts a sub-tile that includes the center pixel. In someoptical systems that are flat and have small error, a sub-tile furtherfrom the ones adjoining the center sub-tile may be used as a referencewithout impacting the overall distortion compensation.

The technology disclosed includes mapping distortion measured oversubstantially the full field of view captured by the image sensor. Threeparameters on which enhanced resolution SIM reconstruction fromregularly structured illumination depend include fringe spacing, fringeangle, and phase displacement of the fringe pattern. These variables arealso referred to as spacing, angle and phase offset of the structuredillumination or grating pattern. The spacing and angle deviations fromthe center tile value may be fit across the full field of view usingpolynomial surfaces. Both quadratic and cubic surfaces have beeninvestigated. Higher order polynomials also may be used.

Both the fringe spacing and fringe angle across the image tile may befit by quadratic surfaces. Sensitivity analysis shows that quadraticsurfaces fit very nearly as well as cubic surfaces. A quadratic surfaceis fit to the following equation (II):

f(x,y)=c0+(c1*x)+(c2*y)+(c3*x*y)+(c4*x ²)+(c5*y ²)  (II)

One implementation of phase estimation adapts the technique proposed byWicker et al. 2013, in their paper titled, “Phase Optimisation forStructured Illumination Microscopy”, section 3. Equations from Lal etal. 2015 titled, “Structured Illumination Microscopy ImageReconstruction Algorithm,” and from Wicker et. al. 2013 help explainWicker phase estimation.

Equation (III) below, taken from Lal et al. 2015 separates three bandsof frequency components: {tilde over (S)}(k) {tilde over (H)}(k); {tildeover (S)}(k−p_(θ)) {tilde over (H)}(k); {tilde over (S)}(k+p_(θ)) {tildeover (H)}(k) from acquired images {tilde over (D)}_(θ,φ) ₁ (k), {tildeover (D)}_(θ,φ) ₂ (k), {tilde over (D)}_(θ,φ) ₃ (k). The mixing matrixuses estimates of the phases φ₁, φ₂, and, φ₃ of images captured using asinusoidal illumination intensity pattern I_(θ,φ)(r), corresponding to apattern angle or orientation 0. Wicker et. al. 2013 refer to phase forn^(th) image at an orientation as φ_(n). If phases are not known withsufficient precision, the unmixing or band separation process willimperfectly separate the spatial frequency components from the observedimages {tilde over (D)}_(θ,φ) ₁ (k), {tilde over (D)}_(θ,φ) ₂ (k),{tilde over (D)}_(θ,φ) ₃ (k) in frequency domain. Practically, the threespatial frequency components {tilde over (S)}(k) {tilde over (H)}(k);{tilde over (S)}(k−p_(θ)) {tilde over (H)}(k); {tilde over (S)}(k+p_(θ)){tilde over (H)}(k) will contain more or less residual information fromother components, as represented by the noise term provided through thefollowing equation (III):

$\begin{matrix}{\begin{bmatrix}\begin{matrix}{{\overset{\sim}{D}}_{\theta,\phi_{1}}(k)} \\{{\overset{\sim}{D}}_{\theta,\phi_{2}}(k)}\end{matrix} \\{{\overset{\sim}{D}}_{\theta,\phi_{3}}(k)}\end{bmatrix} = {{\frac{I_{o}}{2}{M\begin{bmatrix}\begin{matrix}{{\overset{\sim}{S}(k)}{\overset{\sim}{H}(k)}} \\{{\overset{\sim}{S}\left( {k - p_{\theta}} \right)}{\overset{\sim}{H}(k)}}\end{matrix} \\{{\overset{\sim}{S}\left( {k + p_{\theta}} \right)}{\overset{\sim}{H}(k)}}\end{bmatrix}}} + \begin{bmatrix}\begin{matrix}{{\overset{\sim}{N}}_{\theta,\phi_{1}}(k)} \\{{\overset{\sim}{N}}_{\theta,\phi_{2}}(k)}\end{matrix} \\{{\overset{\sim}{N}}_{\theta,\phi_{3}}(k)}\end{bmatrix}}} & ({III})\end{matrix}$ ${{where}M} = \begin{bmatrix}1 & {{- \frac{m}{2}}e^{{- i}\phi_{1}}} & {{- \frac{m}{2}}e^{{+ i}\phi_{1}}} \\1 & {{- \frac{m}{2}}e^{{- i}\phi_{2}}} & {{- \frac{m}{2}}e^{{+ i}\phi_{2}}} \\1 & {{- \frac{m}{2}}e^{{- i}\phi_{3}}} & {{- \frac{m}{2}}e^{{+ i}\phi_{3}}}\end{bmatrix}$

This formulation with three components follows from the Fouriertransform for sine or cosine illumination. A different illuminationfunction may change the equations.

Precise knowledge of the illuminating sinusoidal intensity patternphases may therefore be important. As it is not always possible toprecisely control these phases in experimental setup, it may bedesirable to determine the illumination pattern phases from the acquiredimage data. Wicker et. al. 2013 present a phase estimation technique forSIM data acquired using coherent sinusoidal illumination at a selectedfrequency. Coherent illumination produces good pattern contrast fromfine gratings with a very small illumination peak spacing ‘s’, whichenhances the reconstructed resolution. We retrieve illumination patternphase of the n^(th) image using the illumination pattern's peakfrequency. The illumination pattern's peak frequency is also referred toas Fourier peak.

Equation (IV) below, from Wicker et. al. 2013, presents a generalizedform of equation (II) with acquired images {tilde over (D)}_(n)({rightarrow over (k)}) over frequencies {right arrow over (k)} in thefrequency domain. Each image comprises of three components that arereferred to as {tilde over (C)}⁻¹({right arrow over (k)}), {tilde over(C)}₀({right arrow over (k)}), {tilde over (C)}₊₁({right arrow over(k)}) superimposed with different phases. Note that these threecomponents are the same three components as {tilde over (S)}(k) {tildeover (H)}(k); {tilde over (S)}(k−p_(θ)) {tilde over (H)}(k); {tilde over(H)}(k+p_(θ)) {tilde over (S)}(k) in equation (III).

$\begin{matrix}{{{\overset{\sim}{D}}_{n}\left( \overset{\rightarrow}{k} \right)} = {{{e^{{- i}\phi_{n}}{{\overset{\sim}{C}}_{- 1}\left( \overset{\rightarrow}{k} \right)}} + {{\overset{\rightarrow}{C}}_{0}\left( \overset{\rightarrow}{k} \right)} + {e^{i\phi_{n}}{{\overset{\sim}{C}}_{+ 1}\left( \overset{\rightarrow}{k} \right)}}} = {{\frac{c}{2}e^{{- i}\phi_{n}}{\overset{\sim}{S}\left( {\overset{\rightarrow}{k} + \overset{\rightarrow}{p}} \right)}{\overset{\sim}{h}\left( \overset{\rightarrow}{k} \right)}} + {{\overset{\sim}{S}\left( \overset{\rightarrow}{k} \right)}{\overset{\sim}{h}\left( \overset{\rightarrow}{k} \right)}} + {\frac{c}{2}e^{i\phi_{n}}{\overset{\sim}{S}\left( {\overset{\rightarrow}{k} - \overset{\rightarrow}{p}} \right)}{\overset{\sim}{h}\left( \overset{\rightarrow}{k} \right)}}}}} & ({IV})\end{matrix}$

Note that ‘c’ in equation (IV) is referred to as contrast of theillumination pattern. In the absence of noise, ‘c’ is the same as themodulation factor ‘m’ in mixing matrix M in equation (2). To determineØ_(n), the frequency {right arrow over (k)} in equation (IV) is replacedwith {right arrow over (p)} which is peak frequency of illuminationpattern, resulting in the following equation (V):

$\begin{matrix}{{\phi_{n} \approx {\arg\left\{ {{\overset{\sim}{D}}_{n}\left( \overset{\rightarrow}{p} \right)} \right\}}} = {\arg\left\{ {{\frac{c}{2}e^{{- i}\phi_{n}}{\overset{\sim}{S}\left( {2\overset{\rightarrow}{p}} \right)}{\overset{\sim}{h}\left( \overset{\rightarrow}{p} \right)}} + {{\overset{\sim}{S}\left( \overset{\rightarrow}{p} \right)}{\overset{\sim}{h}\left( \overset{\rightarrow}{p} \right)}} + {\frac{c}{2}e^{i\phi_{n}}{\overset{\sim}{S}(0)}{\overset{\sim}{h}\left( \overset{\rightarrow}{p} \right)}}} \right\}}} & (V)\end{matrix}$

Equation (V) shows that pattern phase Ø_(n) is approximately equal tothe phase of the acquired image {tilde over (D)}_(n)({right arrow over(p)}) over frequency {right arrow over (p)}. This approximate estimationof the pattern phase Ø_(n) may yield good results when three guidelinesare followed. First, the contrast c of the illumination pattern shouldto be sufficiently large. Second, the sample power spectrum shoulddecrease sufficiently fast with growing frequency. When these twoguidelines are followed, equation (V) is dominated by the last term andtherefore, may be simplified to the following equation (VI):

ϕ_(n)≈arg{e ^(iϕ) ^(n) {tilde over (S)}(0){tilde over (h)}({right arrowover (p)})}  (VI)

For any real valued sample, the center frequency {tilde over (S)}(0)will be real valued. Further, if the point spread function (PSF)h({right arrow over (r)}) is real and symmetrical, the optical transferfunction (OTF) {tilde over (h)}({right arrow over (k)}) will be real. AnOTF is a convolution of the point spread function (PSF). A point spreadfunction is the spatial domain version of the optical transfer functionof the imaging system. The name “point spread function” indicates thatall physical optical systems blur (spread) a point of light to somedegree, with the amount of blurring being determined by the quality ofthe optical components. The resolution of the imaging system is limitedby the size of the PSF. For asymmetrical PSFs the phases of the OTFs,should be taken into account.

Third, the OTF at the pattern frequency {tilde over (h)}({right arrowover (p)}) should be sufficiently large to overcome noise. If the OTF istoo small, noise in the acquired image may significantly alter the phasemeasured at {right arrow over (p)}. This phase estimation method cannotbe used for pattern frequencies {right arrow over (p)} outside for thesupport of the detection OTF. For such frequencies, {tilde over(h)}({right arrow over (p)})=0.

An optical system's OTF may be determined experimentally. For example,Lal et al. 2015 compute the OTF by obtaining several images of sampleswith sparsely distributed 100 nm fluorescent microspheres. Intensitydistribution corresponding to more than 100 microspheres were thensuper-imposed and averaged to obtain an approximation for the systemPSF. Fourier transform of this PSF provides an estimate of system OTF.With this background, the phase estimation technique may be applied tosub-tiles.

It may be useful to estimate phase displacement of tiles relative to thefull field of view (FOV), so that measurement of phase in one sub-tilemay be extrapolated to other sub-tiles across the tile. The illuminationpeak angle and illumination peak spacing for the full FOV may beestimated from the illumination peak angle and illumination peak spacingof the sub-tile using the quadratic models presented above. The phasedisplacement may be less regular because it depends pixel geometry ofsub-tiles, which may produce an irregular step function, instead of asmooth function. Phase estimates may be represented using a common frameof reference across sub-tiles of the full FOV image. Sub-tile coordinatespaces may be mapped to a the full FOV coordinate space.

B. Example of Calibration Method for SIM System

Various structural and operational parameters in a SIM optical systemmay adversely impact the quality of the SIM-reconstructed superresolution images. For instance, in any optical system containing lenses(e.g., within lens assembly 280 described above, some other lens that isintegrated into camera system 140), at least one lens may include one ormore structural aberrations, which may produce distortions in imagescaptured by camera system 140. Calculations used in SIM reconstructionmay be sensitive to distortions in source images that are captured usinglenses with aberrations or using an optical assembly 200 having otheraberrations. Increasing the field of view, using most of the lensinstead of a sweet spot in the center, may enhance the susceptibility ofSIM image reconstruction to the distortions caused by aberrations in thelens. Thus, examples described below provide systems and methods fordetecting these lens aberrations; and making adjustments as neededduring image processing to account for such aberrations.

The following description refers to treatment of SIM stacks in a methodof processing. In the present example, each SIM stack includes twelveimages—six images from two channels. For each channel, the set of siximages includes three images taken with reflecting element 270 at threedifferent positions along the linear path LP1 while grating switcher 250is in the first state (e.g., as shown in FIGS. 5A-5B) and another threeimages taken with reflecting element 270 at the same three differentpositions along the linear path LP1 while grating switcher 250 is in thesecond state (e.g., as shown in FIGS. 5C-5D). Thus, the set of siximages for each channel in a SIM stack represents three different phasesfor each of two different grating angles or illumination peak angles.Alternatively, any other suitable number of images may be used to formeach SIM stack, and such images may differ from each other based onparameters other than those identified above.

To account for the imperfections within the optical elements, theprocess of the present example establishes a map of the distortionscreated by these imperfections. Since the imperfections will vary fromsystem 100 to system 100, this mapping process is performed on an ad hocbasis for each system 100, such that each system 100 will have its ownassociated distortion map. While the present example is provided in thecontext of SIM images that have been captured using optical assembly 200and system 100, the process described below may be implemented withvarious other kinds of optical assemblies and systems as will beapparent to those skilled in the art in view of the teachings herein.The process described below is not limited to the context of opticalassembly 200 and system 100.

In order to generate the distortion map, the process analyzes a SIMstack of images using a “sliding window” approach. In this approach, theprocessor only analyzes a square crop or sub-tile of the SIM stack at agiven moment, though the window slides along the field of view of theSIM stack throughout the process and iteratively analyzes severalsub-tiles during the sliding of the window, such that the analyzedsub-tiles ultimately overlap with each other and collectively define asubstantial portion (if not the entirety) of the field of view of theSIM stack. For each sub-tile yielded by the sliding window iterator, thefull width half modulation (FWHM) and parameters are estimated for eachchannel and each angle. All of these estimated parameters and FWHM fromeach sub-tile are saved in a two-dimensional table. In some instances,the goal of the calibration process described herein is to measuremodulation uniformity, spacing and grating angle uniformity (which feedsinto a quadratic surface estimator to provide the distortion models),and phase deviation uniformity.

The above-described process is shown in FIG. 7 . As shown in block 400of FIG. 7 , the process begins with identifying the best-focus SIMstack. As noted above, SIM stacks may be captured at z-positions thatare adjusted in certain increments (e.g., approximately 0.5 μm), suchthat one SIM stack may tend to provide better focus than the other SIMstacks due to the particular z-position of that SIM stack. Conventionalmethods known to those skilled in the art may be used to identify thebest-focus SIM stack from the other SIM stacks. Having identified thebest-focus SIM stack, the process may also define an “estimationwindow,” which may include a window in the center of each image in theSIM stack (i.e., a central region of the view of each image in the SIMstack). This estimation window may be presumed to have the best imagequality for parameter estimation purposes; and may be utilized later inthe process as described below. The size and configuration of theestimation window may be predetermined.

Next, as shown in block 402 of FIG. 7 , the process may apply a slidingwindow iterator to the identified best-focus SIM stack. This slidingwindow iterator may define a sliding window that only views a portion ofthe entire field of view of each image in the SIM stack at a givenmoment. The sliding window iterator may thus scan each image in the SIMstack by moving the sliding window across the entire field of view ofeach image in the SIM stack, processing only a portion of the image at agiven moment. As noted below with reference to FIGS. 8A-8C, the slidingwindow may provide successive views that overlap with each other. Thestride of the sliding window (e.g., distance and speed of movement), thesize of the sliding window, and the configuration of the sliding windowmay be predetermined.

The sliding window iterator may capture data from each image in the SIMstack at any suitable frequency as the sliding window slides across theimage. In some versions, the sliding window may capture data fromregions of the image that overlap with each other. An example of this isshown in FIGS. 8A-8C, which shows a sliding window 500 traversing animage 502. As shown, the position 510 of sliding window 500 at a thirddata capture moment (FIG. 8C) has some spatial overlap 512 with theposition 520 of sliding window at a second data capture moment (FIG.8B); and the position 520 of sliding window 500 at the second datacapture moment has some spatial overlap 522 with the position 530 ofsliding window at a first data capture moment (FIG. 8A). In someimplementations, the spatial overlap 512, 522 may be from approximately60% to approximately 90% of the size of sliding window 500 to provide anoptimal balance of speed and resolution. By way of further example,where sliding window 500 has a size of 512 pixels, the stride of slidingwindow 500 may be up to approximately 200 pixels. Alternatively, thespatial overlap 512, 522 may be less than approximately 60% of the sizeof sliding window 500 or greater than approximately 90% of the size ofsliding window 500.

The sliding window iterator may ultimately yield a plurality ofsub-tiles, representing portions of the base image corresponding to thedata capture moments referred to above. Since each sub-tile is obtainedfrom a corresponding image in a SIM stack (e.g., a stack of twelve baseimages), it may be beneficial to consider the sub-tiles in stackscorresponding to the SIM stack from which the sub-tiles were obtained.Such sub-tiles may be obtained from the same region of the base imagesin the SIM stack, such that the sub-tiles are spatially related. Forshorthand purposes, this collection of spatially related sub-tiles maybe referred to as a sub-tile SIM stack.

After the sliding window iterator has been applied, the process may thenbe applied to each sub-tile yielded by the sliding window iterator. Inparticular, and as shown in block 404 of FIG. 7 , the process mayprovide parameter estimation for each channel and each grating angle ineach sub-tile. The estimated parameters may include modulation, gratingangle, spacing (i.e., local grating spacing or the local SIMillumination pattern periodicity), phase offset, phase deviation, orvarious other parameters. The parameter estimation may include a Fourierdomain algorithm that first performs a rough search for the peak of thegrating pattern (with various pre-processing to enhance the visibilityof the peak). After the rough position is identified, a fine-resolutiongrid-search algorithm may be performed to maximize an objective function(the modulation value at given peak location). The process may alsoestimate the FWHM for each channel and each grating angle in eachsub-tile, as shown in block 406 of FIG. 7 .

Once the parameters have been estimated and the FWHM has been estimatedfor each angle and each grating angle for each sub-tile, these valuesmay be saved for later use, as shown in block 408 of FIG. 7 . In someversions, the values are stored in a two-dimensional table.Alternatively, any other suitable form of storage may be used. In someversions of the process, the process may end at this stage.

In some other versions of the process, the process continues byestimating the center window parameter for each base image in the SIMstack, as shown in block 410 of FIG. 7 . In the present example, thispart of the process is carried out for each base image in the SIM stack;rather than being carried out for each sub-tile as yielded by thesliding window iterator. This center window may be desirable to avoiddistortions that may be more likely to occur near the edges and cornersof each image.

Once the center window parameters have been estimated, the process mayestimate distortion models, as shown in block 412 of FIG. 7 . An exampleof how such distortion models may be estimated will be described belowin further detail with reference to FIG. 9 . The estimated distortionmodel may be used to generate a two-dimensional reference image, asshown in block 414 of FIG. 7 . The two-dimensional reference image mayshow the known regions where the distortions occur in images capturedusing the optical assembly 200 and system 100 at hand. Thetwo-dimensional reference image may thus provide a map that may be laterused to determine where exactly the known distortions are located.

The above-described process may be carried out using a reference opticaltarget, before optical assembly 200 and system 100 are used to captureSIM images of biological samples, etc., in sample container 110 duringnormal use of optical assembly 200 and system 100. In other words, theabove-described process may be carried out during a first use of opticalassembly and system 100, like a calibration procedure. Once thetwo-dimensional reference image has been generated using theabove-described process, when SIM images of biological samples, etc., insample container 110 are later captured during normal use of opticalassembly 200 and system 100, the two-dimensional reference image may befactored into the SIM reconstruction mode, as shown in block 416 of FIG.7 . For instance, the SIM reconstruction process may provide adjustmentsduring reconstruction to account for the known distortions that aremapped out in the two-dimensional reference image. By factoring in theknown distortions as mapped in the two-dimensional reference image, theSIM reconstruction process may ultimately yield more accurate SIMimages.

FIG. 9 shows an example of a process that may be carried out to estimatedistortion models as described above with reference to block 412 of FIG.7 . FIG. 9 thus represents a sub-process that may be carried out duringperformance of the process shown in FIG. 7 . In this example, thedistortion model estimation process subtracts the estimated centerwindow parameter from the two-dimensional tables referred to above, foreach parameter, as shown in block 600 of FIG. 9 . This includesconverting absolute values into biases relative to the center window. Inother words, raw estimated values of various SIM parameters (angle,spacing for each angle and channel combination) are converted intodeviation parameters from the center window. To accomplish this, theprocess may divide the values by the center value (where the deviationis represented as a ratio of the parameter at a specific image subsetlocation vs. that of the same parameter at the center window).Alternatively, the process may subtract the center value (where thedeviation is represented as an offset from the center location).

Then, for each parameter, the process fits the quadratic surfacefunction via a shrinkage estimator in a least squares regression, asshown in block 602 of FIG. 9 . The process then validates the model byensuring that the coefficient of determination (R²) exceeds somethreshold value (e.g., 95), as shown in block 604 of FIG. 9 . Forinstance, a grating distortion metric may be characterized by samplingthe fitted quadratic surface function at specific locations that areorthogonal to the grating direction (e.g., the direction of lines onphase mask 236, 246). While least squares regression is used to fit aquadratic surface function in the present example, any other suitablefitting algorithm may be used. Moreover, any other suitable functionalform (i.e., other than a quadratic surface) may be used.

Returning to the present example, the process then adds the fittedparameter into a distortion model data structure, as shown in block 606of FIG. 9 . At this point, the process completes the estimation of thedistortion model by estimating a phase offset, as shown in block 608 ofFIG. 9 , and storing the estimated phase offset in the data structure(e.g., table), as shown in block 610 of FIG. 9 . During subsequent SIMimage reconstruction, the stored estimated phase offset may beextrapolated to any other window on the SIM image by using the storedtable.

C. Integration of Anti-Stokes Emission in SIM System

When a fluorophore is excited by light, such that the fluorophorefluoresces and thereby emits light, the spectrum of light emitted by thefluorophore is shifted relative to the spectrum of the excitation light.This shift is known as a Stokes shift. This may be attributed to thefact that the energy of a photon emitted by the fluorophore is less thanthe energy of the excitation photon absorbed by the fluorophore. Thisdifference in energy may be caused by energy being lost throughmolecular vibrations occurring when the fluorophore is in the excitedstate. The lost energy may be dissipated as heat to surrounding solventmolecules as they collide with the excited fluorophore. In a Stokesshift scenario, the emitted light may have a wavelength that is longerthan the wavelength of the excitation light.

In some cases, the energy of the photon emitted by the fluorophore isgreater than the energy of the excitation photon absorbed by thefluorophore. Thus, the emitted light may have a wavelength that isshorter than the wavelength of the excitation light. In scenarios wherethe energy of the photon emitted by the fluorophore is greater than theenergy of the excitation photon absorbed by the fluorophore, theemission by the fluorophore may be regarded as an Anti-Stokes emission.

FIG. 10 shows a schematic illustration of Anti-Stokes emission from adye containing a fluorophore. As shown, a pump photon 700 at relativelylong wavelengths is absorbed from higher lying Boltzmann levels within aground state manifold 702 to the bottom of an excited state S1 manifold704, as represented by arrow 706. Subsequent thermalization amongst thevibrational levels of the S1 manifold 704 via phonon-inducedtransitions, followed by emission into the S0 ground state, asrepresented by arrow 708, results in the emission of a higher energyphoton 710, together with subsequent cooling of the dye medium. In thisexample, “S0” and “S1” refer to refer to the singlet electronic statesof the dye molecule. The S nomenclature refers to the total electronicspin of the molecular electronic state. Additionally, dye molecules canalso support triplet spin states, identified by T1, T2, etc. In thepresent example, the focus is on absorption and spontaneous emissionprocesses between the singlet electronic ground state configuration (S0)and the S1 excited singlet configuration. The electronic states of thedye molecule are also coupled to the vibrational modes of the molecule.The multiplicity of vibrational modes within a given electronic statemay be referred to as a vibrational manifold. The term, “vibronic” maybe used to refer to the electronic-vibrational states of a dye molecule.

In some implementations of system 100, an image target at the positionof sample container 110 may include a first dye associated with a firstchannel (e.g., a first color) and a second dye associated with a secondchannel (e.g., a second color). In the present example, the first dye isCoumarin dye, the first channel is a blue channel, the second dye isRhodamine dye, and the second channel is a green channel. Alternatively,any other suitable kinds of dyes or channel colors may be used. In thepresent example, when a light source (e.g., a blue laser in lightemitting assembly 210) emits a light associated with the blue channel,SIM imaging may result in optical detection of blue laser generatedfringes via fluorescence from the Coumarin dye into the blue channel. Inaddition, when a light source (e.g., a green laser in light emittingassembly 210) emits a light associated with the green channel, SIMimaging may result in optical detection of green laser interferencefringes in the green channel via Stokes emission from the Rhodamine dye.However, a low modulation transfer function (MTF) at green emissionwavelengths may result in weak fringe visibility or contrast, making itparticularly difficult to detect the green laser generated fringes atcertain wavelengths (e.g., 600 nm).

To overcome the low contrast that may be observed at green wavelengthsdue to Stokes shifting, it may be desirable to provide an approach thatpermits observation of green laser generated fringes at bluewavelengths. Since the blue channel supports an increased MTF product,the resulting green fringes may be observed with greater fidelity athigher contrast value than may otherwise be obtained. The approach ofgenerating emission at shorter wavelengths than the actual laserexcitation wavelength may rely on the phenomenon of Anti-Stokesfluorescence within a dye molecule. The Anti-Stokes process involvesexcitation of the dye molecule by a photon whose wavelength issubstantially longer than the mean emission wavelength of the dye. Incases where an image target includes Coumarin blue dye, the Coumarinblue dye may be excited by a green laser at longer wavelengths (e.g.,520 nm). As shown in FIG. 10 , green laser excitation of the Coumarindye from higher-lying thermally-excited vibrational modes of theground-state S0 manifold 702 to the bottom of the excited-state S1manifold 704, followed by thermalization within the upper S1 manifold704, leads to wavelength emission 710 much shorter than the initialexcitation wavelength 700. This shorter-wavelength/higher-energyemission 710 may permit observation of green laser generated fringes atblue wavelengths. In other words, a camera tuned to the blue channel mayobserve blue emissions that are generated by a green laser, such thatthe excitation channel and the observation channel may differ from eachother.

V. Miscellaneous

The foregoing description is provided to enable a person skilled in theart to practice the various configurations described herein. While thesubject technology has been particularly described with reference to thevarious figures and configurations, it should be understood that theseare for illustration purposes only and should not be taken as limitingthe scope of the subject technology.

There may be many other ways to implement the subject technology.Various functions and elements described herein may be partitioneddifferently from those shown without departing from the scope of thesubject technology. Various modifications to these implementations maybe readily apparent to those skilled in the art, and generic principlesdefined herein may be applied to other implementations. Thus, manychanges and modifications may be made to the subject technology, by onehaving ordinary skill in the art, without departing from the scope ofthe subject technology. For instance, different numbers of a givenmodule or unit may be employed, a different type or types of a givenmodule or unit may be employed, a given module or unit may be added, ora given module or unit may be omitted.

Some versions of the examples described herein may be implemented usinga computer system, which may include at least one processor thatcommunicates with a number of peripheral devices via bus subsystem.These peripheral devices may include a storage subsystem including, forexample, memory devices and a file storage subsystem, user interfaceinput devices, user interface output devices, and a network interfacesubsystem. The input and output devices may allow user interaction withthe computer system. The network interface subsystem may provide aninterface to outside networks, including an interface to correspondinginterface devices in other computer systems. User interface inputdevices may include a keyboard; pointing devices such as a mouse,trackball, touchpad, or graphics tablet; a scanner; a touch screenincorporated into the display; audio input devices such as voicerecognition systems and microphones; and other types of input devices.In general, use of the term “input device” is intended to include allpossible types of devices and ways to input information into computersystem.

User interface output devices may include a display subsystem, aprinter, a fax machine, or non-visual displays such as audio outputdevices. The display subsystem may include a cathode ray tube (CRT), aflat-panel device such as a liquid crystal display (LCD), a projectiondevice, or some other mechanism for creating a visible image. Thedisplay subsystem may also provide a non-visual display such as audiooutput devices. In general, use of the term “output device” is intendedto include all possible types of devices and ways to output informationfrom computer system to the user or to another machine or computersystem.

A storage subsystem may store programming and data constructs thatprovide the functionality of some or all of the modules and methodsdescribed herein. These software modules may be generally executed bythe processor of the computer system alone or in combination with otherprocessors. Memory used in the storage subsystem may include a number ofmemories including a main random access memory (RAM) for storage ofinstructions and data during program execution and a read only memory(ROM) in which fixed instructions are stored. A file storage subsystemmay provide persistent storage for program and data files, and mayinclude a hard disk drive, a floppy disk drive along with associatedremovable media, a CD-ROM drive, an optical drive, or removable mediacartridges. The modules implementing the functionality of certainimplementations may be stored by file storage subsystem in the storagesubsystem, or in other machines accessible by the processor.

The computer system itself may be of varying types including a personalcomputer, a portable computer, a workstation, a computer terminal, anetwork computer, a television, a mainframe, a server farm, awidely-distributed set of loosely networked computers, or any other dataprocessing system or user device. Due to the ever-changing nature ofcomputers and networks, the example of the computer system describedherein is intended only as a specific example for purposes ofillustrating the technology disclosed. Many other configurations of acomputer system are possible having more or less components than thecomputer system described herein.

As an article of manufacture, rather than a method, a non-transitorycomputer readable medium (CRM) may be loaded with program instructionsexecutable by a processor. The program instructions when executed,implement one or more of the computer-implemented methods describedabove. Alternatively, the program instructions may be loaded on anon-transitory CRM and, when combined with appropriate hardware, becomea component of one or more of the computer-implemented systems thatpractice the methods disclosed.

Underlined and/or italicized headings and subheadings are used forconvenience only, do not limit the subject technology, and are notreferred to in connection with the interpretation of the description ofthe subject technology. All structural and functional equivalents to theelements of the various implementations described throughout thisdisclosure that are known or later come to be known to those of ordinaryskill in the art are expressly incorporated herein by reference andintended to be encompassed by the subject technology. Moreover, nothingdisclosed herein is intended to be dedicated to the public regardless ofwhether such disclosure is explicitly recited in the above description.

It should be appreciated that all combinations of the foregoing conceptsand additional concepts discussed in greater detail below (provided suchconcepts are not mutually inconsistent) are contemplated as being partof the inventive subject matter disclosed herein. In particular, allcombinations of claimed subject matter appearing at the end of thisdisclosure are contemplated as being part of the inventive subjectmatter disclosed herein.

1.-20. (canceled)
 21. A method including: receiving a plurality ofimages captured using structured illumination microscopy (SIM) in anoptical system, each image of the plurality of images having a firstfield of view; defining a window, the window defining a second field ofview representing a portion of the first field of view such that thesecond field of view is smaller than the first field of view; moving thewindow in relation to each image of a plurality of images; capturing aplurality of sub-tiles from each image of the plurality of images whilemoving the window in relation to each image of the plurality of images,each sub-tile of the plurality of plurality of sub-tiles representing aportion of the corresponding image of the plurality of images, theportion represented by each sub-tile of the plurality of sub-tiles beingdefined by the second field of view at a position corresponding to amoment at which the sub-tile of the plurality of sub-tiles is captured;estimating parameters associated with each sub-tile of the plurality ofsub-tiles, the parameters comprising an angle, a spacing, and a phaseoffset; and storing the estimated parameters in a predetermined format.22. The method of claim 21, the parameters further including a parameterselected from the group consisting of modulation and phase deviation.23. The method of claim 21, further comprising: estimating a full widthat half maximum (FWHM) value associated with each sub-tile of theplurality of sub-tiles; and storing the FWHM values in a predeterminedformat.
 24. The method of claim 21, further comprising: estimating acenter window parameter, the center window parameter corresponding to acentral region within the first field of view; and estimating adistortion model based at least in part on a combination of theestimated parameters stored in the predetermined format and theestimated center window parameter, the estimating the distortion modelincluding subtracting the estimated center window parameter from theestimated parameters stored in the predetermined format.
 25. The methodof claim 24, further comprising: capturing a subsequent plurality ofimages using SIM in the optical system; and generating a high-resolutionimage based at least in part on the plurality of images, the generatingthe high-resolution image including adjusting data from the subsequentplurality of images based at least in part on the estimated distortionmodel.
 26. A method including: receiving a plurality of images capturedusing structured illumination microscopy (SIM) in an optical system,each image of the plurality of images having a first field of view;defining a window, the window defining a second field of viewrepresenting a portion of the first field of view such that the secondfield of view is smaller than the first field of view; moving the windowin relation to each image of a plurality of images; capturing aplurality of sub-tiles from each image of the plurality of images whilemoving the window in relation to each image of the plurality of images,each sub-tile of the plurality of plurality of sub-tiles representing aportion of the corresponding image of the plurality of images, theportion represented by each sub-tile of the plurality of sub-tiles beingdefined by the second field of view at a position corresponding to amoment at which the sub-tile of the plurality of sub-tiles is captured;estimating a first set of parameters for a first sub-tile of theplurality of sub-tiles, the first set of parameters comprising an angleand a spacing; mapping the first set of parameters to a second sub-tileof the plurality of sub-tiles; estimating a phase offset for the secondsub-tile of the plurality of sub-tiles using the mapped first set ofparameters; and storing the mapped first set of parameters and theestimated phase offset for the second sub-tile in a predeterminedformat.
 27. The method of claim 26, the parameters further including aparameter selected from the group consisting of modulation, phaseoffset, and phase deviation.
 28. The method of claim 26, furthercomprising: estimating a full width at half maximum (FWHM) valueassociated with each sub-tile of the plurality of sub-tiles; and storingthe FWHM values in a predetermined format.
 29. The method of claim 26,further comprising: estimating a center window parameter, the centerwindow parameter corresponding to a central region within the firstfield of view; and estimating a distortion model based at least in parton a combination of the estimated parameters stored in the predeterminedformat and the estimated center window parameter, the estimating thedistortion model including subtracting the estimated center windowparameter from the estimated parameters stored in the predeterminedformat.
 30. The method of claim 29, further comprising: capturing asubsequent plurality of images using SIM in the optical system; andgenerating a high-resolution image based at least in part on theplurality of images, the generating the high-resolution image includingadjusting data from the subsequent plurality of images based at least inpart on the estimated distortion model.
 31. An apparatus comprising: afirst optical assembly to emit structured illumination toward a target,the first optical assembly including: a light emitting assembly, a firstphase mask to impart a first pattern to light emitted by the lightemitting assembly, a second phase mask to impart a second pattern tolight emitted by the light emitting assembly, and a phase adjustmentassembly to adjust a phase of light structured by the first phase maskand the second phase mask; a second optical assembly, the second opticalassembly including an image sensor to capture images of the target asilluminated by the first optical assembly; and a processor, theprocessor to perform the following: receive a plurality of imagescaptured using the image sensor, each image of the plurality of imageshaving a first field of view, define a window, the window defining asecond field of view representing a portion of the first field of viewsuch that the second field of view is smaller than the first field ofview, move the window in relation to each image of a plurality ofimages, capture a plurality of sub-tiles from each image of theplurality of images while moving the window in relation to each image ofthe plurality of images, each sub-tile of the plurality of plurality ofsub-tiles representing a portion of the corresponding image of theplurality of images, the portion represented by each sub-tile of theplurality of sub-tiles being defined by the second field of view at aposition corresponding to a moment at which the sub-tile of theplurality of sub-tiles is captured, estimate a first set of parametersfor a first sub-tile of the plurality of sub-tiles, the first set ofparameters comprising an angle and a spacing, map the first set ofparameters to a second sub-tile of the plurality of sub-tiles, estimatea phase offset for the second sub-tile of the plurality of sub-tilesusing the mapped first set of parameters, and store the mapped first setof parameters and the estimated phase offset for the second sub-tile ina predetermined format.
 32. The apparatus of claim 31, the first set ofparameters further including a parameter selected from the groupconsisting of modulation, phase offset, and phase deviation.
 33. Theapparatus of claim 31, the processor further configured to: estimate afull width at half maximum (FWHM) value associated with each sub-tile ofthe plurality of sub-tiles, and store the estimated FWHM values in apredetermined format.
 34. The apparatus of claim 31, the processorfurther to: estimate a center window parameter, the center windowparameter corresponding to a central region within the first field ofview, and estimate a distortion model based at least in part on acombination of the estimated parameters stored in the predeterminedformat and the estimated center window parameter, the estimating thedistortion model including subtracting the estimated center windowparameter from the estimated parameters stored in the predeterminedformat.
 35. The apparatus of claim 34, the processor further to: receivea subsequent plurality of images captured using the image sensor, andgenerate a high-resolution image based at least in part on the pluralityof images, the generating the high-resolution image including adjustingdata from the subsequent plurality of images based at least in part onthe estimated distortion model.
 36. An apparatus comprising: an opticalsystem, the optical system being operable to capture a plurality ofimages using structured illumination microscopy (SIM); and a processor,the processor to perform the following: receive a plurality of imagescaptured using SIM in the optical system, each image of the plurality ofimages having a first field of view, define a window, the windowdefining a second field of view representing a portion of the firstfield of view such that the second field of view is smaller than thefirst field of view, move the window in relation to each image of aplurality of images, capture a plurality of sub-tiles from each image ofthe plurality of images while moving the window in relation to eachimage of the plurality of images, each sub-tile of the plurality ofplurality of sub-tiles representing a portion of the corresponding imageof the plurality of images, the portion represented by each sub-tile ofthe plurality of sub-tiles being defined by the second field of view ata position corresponding to a moment at which the sub-tile of theplurality of sub-tiles is captured, estimate parameters associated witheach sub-tile of the plurality of sub-tiles, the parameters comprisingan angle, a spacing, and a phase offset, and store the estimatedparameters in a predetermined format.
 37. The apparatus of claim 36, thefirst set of parameters further including a parameter selected from thegroup consisting of modulation and phase deviation.
 38. The apparatus ofclaim 36, the processor further configured to: estimate a full width athalf maximum (FWHM) value associated with each sub-tile of the pluralityof sub-tiles, and store the estimated FWHM values in a predeterminedformat.
 39. The apparatus of claim 36, the optical system comprising: afirst optical assembly to emit structured illumination toward a target,the first optical assembly including: a light emitting assembly, a firstphase mask to impart a first pattern to light emitted by the lightemitting assembly, a second phase mask to impart a second pattern tolight emitted by the light emitting assembly, and a phase adjustmentassembly to adjust a phase of light structured by the first phase maskand the second phase mask, and a second optical assembly, the secondoptical assembly including an image sensor to capture images of thetarget as illuminated by the first optical assembly.
 40. The apparatusof claim 36, the processor further to: estimate a center windowparameter, the center window parameter corresponding to a central regionwithin the first field of view, estimate a distortion model based atleast in part on a combination of the estimated parameters stored in thepredetermined format and the estimated center window parameter, theestimating the distortion model including subtracting the estimatedcenter window parameter from the estimated parameters stored in thepredetermined format, receive a subsequent plurality of images capturedusing SIM in the optical system, and generate a high-resolution imagebased at least in part on the plurality of images, the generating thehigh-resolution image including adjusting data from the subsequentplurality of images based at least in part on the estimated distortionmodel.